Inspection device, inspection method and storage medium

ABSTRACT

The acquisition unit  41 B acquires a target image indicating a target object of inspection. The detection unit  42 B detects, on the basis of a group of images each of which indicates the target object in a normal state, the target image that indicates the target object that is not in the normal state among the target images that the acquisition unit  41 B acquires.

TECHNICAL FIELD

The present invention relates to a technical field of an inspectiondevice, an inspection method and a storage medium for inspection.

BACKGROUND ART

There is an endoscope system for automatically detecting a predeterminedarea in the lumen of an organ. For example, Patent Literature 1discloses an endoscope system which displays on a display unit aposition information of the detected area of interest based on anendoscope image at the time of detecting the area of interest whichincludes at least one of a lesion parts or a normal part in the lumen.The endoscope system described above measures the insertion length ofthe endoscope and generates position information based on the measuredinsertion length.

PRIOR ART DOCUMENTS Patent Literature

Patent Literature 1: International Publication WO2018/179991

SUMMARY OF THE INVENTION Problem to be Solved by the Invention

In the case like an endoscope system that an image indicating a lesionpart to be inspected is selected by visual judgment from a plurality ofimages continuously obtained, there is a possibility that an importantinspection may be overlooked. On the other hand, although PatentLiterature 1 discloses a technique of detecting an area of interestincluding at least one of a lesion parts or a normal part in the lumenby template matching or the like, it is silent on a technique foraccurately detecting an image indicating that a target object is not ina normal state.

In view of the above-described issues, it is therefore an example objectof the present disclosure to provide an inspection device, an inspectionmethod and a storage medium capable of suitably detecting an imageindicating that a target object is not in a normal state.

Means for Solving the Problem

One mode of the inspection device is an inspection device including: anacquisition unit configured to acquire a target image which indicates atarget object of inspection; and a detection unit configured to detect,on a basis of a group of images each of which indicates the targetobject in a normal state, the target image that indicates the targetobject that is not in the normal state among the target images that theacquisition unit acquires.

One mode of the inspection method is an inspection method executed by aninspection device, the inspection method including: acquiring a targetimage which indicates a target object of inspection; and detecting, on abasis of a group of images each of which indicates the target object ina normal state, the target image that indicates the target object thatis not in the normal state among the acquired target images.

One mode of the storage medium is a non-transitory computer-readablestorage medium storing a program executed by a computer, the programcausing the computer to function as: an acquisition unit configured toacquire a target image which indicates a target object of inspection;and a detection unit configured to detect, on a basis of a group ofimages each of which indicates the target object in a normal state, thetarget image that indicates the target object that is not in the normalstate among the target images that the acquisition unit acquires.

Effect of the Invention

An example advantage according to the present invention is to suitablydetect, from acquired images, a target image indicating that a targetobject is not in a normal state.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates a schematic configuration of an endoscopic inspectionsystem according to a first embodiment.

FIG. 2 illustrates a hardware configuration of an inspection deviceaccording to the first embodiment.

FIG. 3 is a functional block diagram of the inspection device accordingto the first embodiment.

FIG. 4 illustrates a photographed image that was not determined as ananomaly image.

FIG. 5 illustrates a photographed image determined as an anomaly image.

FIG. 6 is an example of a flowchart showing a processing procedure ofthe inspection device according to the first embodiment.

FIG. 7 illustrates a hardware configuration of an inspection deviceaccording to a second embodiment.

FIG. 8 is a functional block diagram of an inspection device accordingto the second embodiment.

FIG. 9 is a display example of a pathological tissue image.

FIG. 10 illustrates the restoration error when a pathological tissueimage is inputted to an auto encoder.

FIG. 11 is an example of a flowchart showing a processing procedure ofthe inspection device according to the second embodiment.

FIG. 12 is a functional block diagram of the inspection device accordingto a third embodiment.

DESCRIPTION OF EXAMPLE EMBODIMENTS

Hereinafter, with reference to the drawings, an example embodiment ofthe inspection device, the inspection method and the storage medium willbe described.

First Example Embodiment

First, an example embodiment relating to an endoscopic inspection systemwill be described.

(1-1) Configuration

FIG. 1 illustrates a schematic configuration of an endoscopic inspectionsystem 100. As shown in FIG. 1, the endoscopic inspection system 100mainly includes an inspection device 1, a display device 2, and anendoscope 3 connected to the inspection device 1.

The inspection device 1 acquires from the endoscope 3 an image (alsoreferred to as a “photographed image Ic”), which the endoscope 3photographs (captures) in time series, and displays the photographedimage Ic and the associated information on the display device 2.According to the present example embodiment, the inspection device 1detects a photographed image Ic indicating a part (i.e., anomalous part)in an abnormal state from photographed images Ic each of which indicatesthe lumen of the subject's organ to be photographed such as a largebowel and then regards the detected photographed image Ic as a landmark.Details of this process will be described later. Hereinafter, as arepresentative example, a description will be given of the process inthe endoscopy relating to the large bowel.

The display device 2 is a display or the like for displaying informationbased on the display information supplied from the inspection device 1.

The endoscope 3 is the equipment which photographs the lumen of thelarge bowel by inserting it in the subject's large bowel. The endoscope3 mainly includes a connecting portion 31 for connecting with theinspection device 1, an operation unit 32 for inspector to perform apredetermined input, a shaft 33 to be inserted into the lumen and havingflexibility, and a pointed end unit 34 having a built-in photographingunit such as an ultra-small image pickup device.

FIG. 2 illustrates the hardware configuration of the inspection device1. The inspection device 1 mainly includes a processor 11, a memory 12,an interface 13, an input unit 14 and a light source unit 15. Each ofthese elements is connected via a data bus 19.

The processor 11 executes a predetermined process by executing a programor the like stored in the memory 12. The processor 11 is a processorsuch as a CPU (Central Processing Unit) or a GPU (Graphics ProcessingUnit).

The memory 12 includes various memories such as a RAM (Random AccessMemory), a ROM (Read Only Memory), and a flash memory. Further, thememory 12 stores a program for the inspection device 1 to execute eachprocess according to the present example embodiment. The memory 12 isalso used as a work memory. Further, the memory 12 functionally includesa parameter information storage unit 20, a landmark position informationstorage unit 21 and an attention part information storage unit 22.Details of these storage units will be described later.

The interface 13 includes an interface for connecting the inspectiondevice 1 and the display device 2, and an interface for connecting theinspection device 1 and the endoscope 3. For example, the interface 13supplies display information generated by the processor 11 to thedisplay device 2. Further, the interface 13 supplies the light generatedby the light source unit 15 to the endoscope 3. The interface 13 alsoprovides the processor 11 with an electrical signal indicative of thephotographed image Ic supplied from the endoscope 3.

The input unit 14 generates an input signal based on the operation bythe inspector. The input unit 14 is, for example, a button, a touchpanel, a remote controller, a voice input device, or the like. The lightsource unit 15 generates light for supplying to the pointed end unit 34of the endoscope 3. The light source unit 15 may also incorporate a pumpor the like for delivering water and air to be supplied to the endoscope3.

Next, the parameter information storage unit 20, the landmark positioninformation storage unit 21, and the attention part information storageunit 22 will be described.

The parameter information storage unit 20 stores parameter informationindicating parameters for configuring the auto encoder (auto-encoder).The above-described auto encoder is a neural network which is learned tooutput data that is reproduced input data in the form of an image and isused to detect a photographed image Ic (also referred to as an “anomalyimage Ica”) that indicates an anomalous part. The parameter informationstored in the parameter information storage unit 20 is the informationindicative of the parameter regarding the weight of the network of theauto encoder and the like, and is obtained by learning the auto encoderusing a plurality of images (group of images) obtained by photographing(capturing) the inspection target (here, the lumen of the large bowel)in the normal state. Examples of the anomaly image Ica includes aphotographed image Ic which indicates a part where the irritation isoccurring, a part corresponding to an operation mark or other, a partcorresponding to folds or protrusions or a lesion (diseased) part suchas cancers.

The landmark position information storage unit 21 stores information(also referred to as “landmark position information”) indicating theposition of the landmark in the interior of the large bowel that is aphotographing target. In the present example embodiment, the processor11 regards a part within a photographing range of the anomaly image Icaas a landmark. The processor 11 stores information on the landmark aslandmark position information in the landmark position informationstorage unit 21. As will be described later, the landmark functions notonly as a mark indicating the position in the entire lumen of the largebowel to be inspected but also as a mark indicating the candidate to beinspected.

The landmark position information includes, for example, an anomalyimage Ica detected by a detector, information (also referred to as“feature information Ifa”) indicating the features of the anomaly imageIca and an identification number of the landmark (e.g., a serialnumber). The feature information Ifa may be, for example, a compressedfeature (latent variable) of the anomaly image Ica calculated by theauto encoder or may be the restoration error.

The attention part information storage unit 22 stores information (alsoreferred to as “attention part information”) indicating the attentionpart that is a part to be paid attention to in the inspection other thanthe landmark. As will be described later, the attention part correspondsto a part specified by the inspector or a part detected through an imageanalysis program for detecting a lesion (defective) part or the like.The attention part information includes, for example, a photographedimage Ic indicating an attention point (also referred to as “attentionpart image Icr”), an identification number (e.g., a serial number) ofthe attention part, and an identification number of a landmark existingbefore and after the attention part. The attention part information mayfurther include feature information of the attention part image Icrobtained by inputting the attention part image Icr to theabove-described auto encoder.

(1-2) Functional Block

FIG. 3 is a functional block diagram of the inspection device 1. Asshown in FIG. 3, the processor 11 of the inspection device 1functionally includes an acquisition unit 41, a detection unit 42, aposition information generation unit 43, a position determination unit44 and an output unit 45.

The acquisition unit 41 acquires a photographed image Ic, which theendoscope 3 captures in time series, at predetermined intervals via theinterface 13. Then, in the insertion process of inserting the endoscope3 into the lumen the acquisition unit 41, the acquisition unit 41supplies the acquired photographed image Ic to the detection unit 42. Inthe discharge process of the endoscope 3, the acquisition unit 41supplies the acquired photographed image Ic to the positiondetermination unit 44. For example, on the basis of an input (e.g.,input indicating the start of the insertion and the completion thereof)by the operation unit 32 of the endoscope 3 or by the input unit 14 ofthe inspection device 1, the acquisition unit 41 may determine whetherit is in the insertion process or in the discharge process of theendoscope 3. In another example, the acquisition unit 41 may make theabove-described determination based on the image recognition processingby use of the photographed image Ic or the output of the sensor attachedto the endoscope 3.

Further, when input information (also referred to as “input informationIi”) generated by the operation unit 32 for specifying the attentionpart is received from the endoscope 3, the acquisition unit 41 suppliesthe position information generation unit 43 with the input informationIi and the photographed image Ic, which is received from the endoscope 3at the same timing as the input information Ii. Further, the acquisitionunit 41 supplies the photographed image Ic to the output unit 45regardless of being in the insertion process or in the discharge processof the endoscope 3. Thereby, the photographed image Ic acquired by theacquisition unit 41 is immediately displayed on the display device 2.

The detection unit 42 detect the anomaly image Ica showing an anomalouspart in the lumen among the photographed images Ic. In this case, thedetection unit 42 configures an auto encoder by referring to theparameter information storage unit 20. Then, the detection unit 42calculates an error (so-called restoration error) between the outputtedimage obtained by inputting the photographed image Ic supplied from theacquisition unit 41 to the auto encoder and the inputted photographedimage Ic. In this case, 42, as the restoration error, for example, theaverage value or any other representative value of the difference of thepixel values for each pixel between the output image and thephotographed image Ic are calculated. The detection unit 42 determinesthat the inputted photographed image Ic is the anomaly image Ica if thecalculated restoration error is equal to or larger than a predeterminedthreshold value. The threshold value described above is predeterminedthrough experimental trials, for example. When detecting the anomalyimage Ica, the detection unit 42 supplies the detected anomaly image Icaand the feature information Ifa indicating the features of the anomalyimage Ica generated by the auto encoder (e.g., latent variable orrestoration error, etc.) to the position information generation unit 43.

When receiving the anomaly image Ica and the feature information Ifafrom the detection unit 42, the position information generation unit 43generates landmark position information including the anomaly image Icaand the feature information Ifa, and stores the generated landmarkposition information in the landmark position information storage unit21.

Further, when the input information Ii and the photographed image Ic aresupplied from the acquisition unit 41, the position informationgeneration unit 43 determines that an attention part is designated.Therefore, in this case, the position information generation unit 43regards the photographed image Ic supplied together with the inputinformation Ii as the attention part image Icr and stores the attentionpart information including the attention part image Icr in the attentionpart information storage unit 22. Further, when the position informationgeneration unit 43 executes an image analysis program for detecting atarget to be inspected such as a lesion (defective) part with respect tothe photographed image Ic that is acquired by the acquisition unit 41,the position information generation unit 43 regards the photographedimage Ic indicating the target to be inspected detected by the imageanalysis program as the attention part image Icr. Then, the positioninformation generation unit 43 stores the attention part informationincluding the attention part image Icr in the attention part informationstorage unit 22.

In some example embodiment, the position information generation unit 43may associate the attention part information with at least one of thelandmark position information generated immediately before the attentionpart image Icr is detected or the landmark position informationregarding the landmark generated immediately after the attention partimage Icr is detected. For example, the position information generationunit 43 includes at least one of the identification number of thelandmark detected immediately before the attention part image Icr isdetected or the identification number of the landmark detectedimmediately after the attention part image Icr is detected in theattention part information. Thus, the position information generationunit 43 can generate the attention part information by which therelative positional relation between the position of the landmarkspecified by the landmark position information and the attention part ofinterest can be suitably specified. In addition, when generating theattention part information, the position information generation unit 43may include the feature information obtained by inputting the attentionpart image Icr to the above-described auto encoder in the attention partinformation.

When the position determination unit 44 receives the photographed imageIc from the acquisition unit 41, the position determination unit 44determines whether or not it has received the photographed image Icwhich indicates the same part as the landmark stored in the landmarkposition information storage unit 21. For example, the positiondetermination unit 44 cross-checks (collates) the feature informationobtained by inputting the photographed image Ic to the auto encoderconfigured with reference to the parameter information storage unit 20with the feature information Ifa included in the landmark positioninformation stored in the landmark position information storage unit 21.Then, when there is the landmark position information including thefeature information Ifa whose deviation (error) from the featureinformation of the photographed image Ic is equal to or smaller than apredetermined degree, the position determination unit 44 determines thatthe endoscope 3 is currently photographing the position indicated by thelandmark position information. Therefore, in this case, the positiondetermination unit 44 supplies the anomaly image Ica included in thetarget landmark position information to the output unit 45.

Further, when the position determination unit 44 receives thephotographed image Ic from the acquisition unit 41, the positiondetermination unit 44 determines whether or not it has received thephotographed image Ic that captures the same part as the attention partstored in the attention part information storage unit 22. For example,the position determination unit 44 cross-checks the feature informationof the photographed image Ic supplied from the acquisition unit 41 withthe feature information of the attention part image Icr included in theattention part information stored in the attention part informationstorage unit 22. Then, the position determination unit 44 determinesthat the pointed end unit 34 is capturing the position indicated by theattention part image Icr if there is the attention part image Icr whosefeature information is deviated from the feature information of thephotographed image Ic supplied from the acquisition unit 41 by a degreeequal to or smaller than a predetermined degree. Therefore, in thiscase, the position determination unit 44 supplies the attention partimage Icr to the output unit 45.

The output unit 45 immediately displays the photographed image Ic whichthe acquisition unit 41 acquires on the display device 2 by supplying avideo signal indicating the photographed image Ic supplied from theacquisition unit 41 to the display device 2. Further, when receiving theanomaly image Ica from the position determination unit 44, the outputunit 45 displays the anomaly image Ica together with the photographedimage Ic which the acquisition unit 41 acquires on the display device 2and further outputs information prompting confirmation of the currentphotographing position. The “outputs information prompting confirmation”may be realized by displaying information by a character or a figure orby outputting a voice. Thereby, the output unit 45 enables the inspectorto suitably recognize the presence of the part determined to be ananomalous part and suitably suppresses the inspector from overlookingthe part where inspection is necessary. In some example embodiments, theoutput unit 45 may further display the total number of landmarks storedin the landmark position information storage unit 21 and the number ofthe detected order of the detected landmark in the insertion process onthe display device 2. This enables the inspector to appropriatelyrecognize the current approximate photographing position in the entirelumen of the large bowel to be inspected.

Similarly, when receiving the attention part image Icr from the positiondetermination unit 44, the output unit 45 displays the attention partimage Icr together with the photographed image Ic acquired by theacquisition unit 41 on the display device 2 and outputs informationprompting confirmation of the current photographed position. In thiscase, in some example embodiments, the output unit 45 may furtherdisplay the identification number of the landmarks before and after theattention part of interest on the display device 2 by referring to thecorresponding attention part information from the attention partinformation storage unit 22. Thereby, the output unit 45 enable theinspector to suitably recognize the relative position of the attentionpart of interest with respect to the landmark.

If the output unit 45 receives the photographed image Ic indicating thesame part as the landmark or the attention part from the endoscope 3 andthe length of the duration of receiving the photographed image Icindicating the same part as the target landmark or the target attentionpart is equal to or shorter than the threshold value, the output unit 45may output a predetermined warning. The above-described threshold valueis set to be a time length for which it is determined that theconfirmation by the inspector has not been performed for the targetlandmark or the target attention part. Thereby, it is possible to morereliably suppress the inspector from overlooking the part whereinspection is necessary.

(1-3) Specific examples of anomaly images

Here, specific examples of determination relating to the anomaly imageIca by the detection unit 42 will be described with reference to FIGS. 4and 5.

FIG. 4 illustrates a photographed image “Ic1” that was not determined asan anomaly image Ica by the detection unit 42. Further, FIG. 5illustrates a photographed image “Ic2” which is determined to be ananomaly image Ica by the detection unit 42.

In the photographed image Ic1 shown in FIG. 4, there is no part wherethe irritation is generated, a part where an operation mark or othercuts is generated, a part where a fold or protrusion is generated nor alesion (diseased) part such as a cancer, or the like. The restorationerror calculated at the time when the photographed image Ic1 is inputtedto the auto encoder configured by referring to the parameter informationstored in the parameter information storage unit 20 is smaller than thethreshold value. Therefore, in this case, the detection unit 42determines that the photographed image Ic1 is not the anomaly image Ica.

On the other hand, as indicated by the broken line frame 50 in FIG. 5,there is a protrusion part in the photographed image Ic2. Then, therestoration error calculated at the time when the photographed image Ic2is inputted to the above-described auto encoder is equal to or largerthan the threshold value. Therefore, in this case, the detection unit 42determines that the photographed image Ic2 is the anomaly image Ica.

In this way, by using the auto encoder learned based only on the imagesobtained by photographing the lumen in the normal state, the detectionunit 42 can suitably detect the part where the irritation is occurring,the part where the operation mark or other cuts are occurring, the partwhere the fold or protrusion is occurring and the lesion part such as acancer as a landmark. In addition, by using an auto encoder learnedbased only on the image which captured the lumen in the normal state,there is also such an advantage that a large amount of learning data canbe prepared with little labor, because the detailed annotation by thephysiologist for generating the learning data becomes unnecessary.

(1-4) Process Flow

FIG. 6 is an example of a flowchart showing a processing procedure to beperformed by the inspection device 1 in endoscopy according to the firstexample embodiment.

First, the acquisition unit 41 of the inspection device 1 determineswhether or not the pointed end unit 34 of the endoscope 3 is insertedinto the lumen of the large bowel to be inspected (step S101). Then,when the pointed end unit 34 is inserted into the large bowel (stepS101; Yes), the acquisition unit 41 acquires the photographed image Icat predetermined intervals from the endoscope 3 (step S102). Further,the acquisition unit 41 lets the display device 2 immediately displaythe photographed image Ic acquired in the time series by supplying theacquired photographed image Ic to the output unit 45.

Next, the detection unit 42 of the inspection device 1 inputs thephotographed image Ic to the auto encoder configured with reference tothe parameter information storage unit 20 (step S103). Thereby, thedetection unit 42 calculates the restoration error of the photographedimage Ic. Then, on the basis of the calculated restoration error, thedetection unit 42 determines whether or not the photographed image Iccorresponds to the anomaly image Ica (step S104).

When the photographed image Ic corresponds to the anomaly image Ica(step S104; Yes), the position information generation unit 43 of theinspection device 1 generates the landmark position information andstores the landmark position information in the landmark positioninformation storage unit 21 (step S105). For example, when therestoration error described above is equal to or larger than apredetermined threshold value, the position information generation unit43 generates and stores the landmark position information including theidentification number of the landmark, the target anomaly image Ica andits feature information Ifa. On the other hand, when the photographedimage Ic does not correspond to the anomaly image Ica (step S104; No),the process proceeds to step S106.

Furthermore, the position information generation unit 43 determineswhether or not the attention part has been designated or detected (stepS106). For example, when the acquisition unit 41 acquires the inputinformation Ii and the photographed image Ic relating to the designationof the attention part from the endoscope 3, the position informationgeneration unit 43 determines that the attention part has beendesignated. In another example, the position information generation unit43 determines that the attention part has been detected when the imageanalysis program that detects the photographed image Ic indicating theinspection target has detected the photographed image Ic indicating theinspection target. When the attention part has been designated ordetected (Step S106; Yes), the position information generation unit 43generates the attention part information including the targetphotographed image Ic as the attention part image Icr, and stores theattention part information in the attention part information storageunit 22 (Step S107).

Next, the acquisition unit 41 determines whether or not it is in thedischarge process for discharging the endoscope 3 from the large bowelto be inspected (step S108). Then, when the acquisition unit 41determines that it is not in the discharge process (step S108; No),i.e., when it is determined that it is still in the insertion process,the acquisition unit 41 returns the process to the step S102.

On the other hand, when the acquisition unit 41 determines that it is inthe discharge process (step S108; Yes), the acquisition unit 41 acquiresthe photographed image Ic from the endoscope 3 (step S109). Further, theacquisition unit 41 immediately displays the acquired photographed imageIc on the display device 2 by supplying the acquired photographed imageIc to the output unit 45.

Then, the position determination unit 44 of the inspection device 1determines whether or not the position (i.e., the shooting range of thephotographed image Ic) indicated by the photographed image Ic acquiredat step S109 corresponds to the landmark or the attention part (stepS110). In this case, if there is any landmark position informationincluding the feature information Ifa whose deviation from the featureinformation of the photographed image Ic acquired at step S109 issmaller than a predetermined threshold value, the position determinationunit 44 determines that the position indicated by the photographed imageIc acquired at step S109 corresponds to the landmark. In the same way,if there is any attention part information including the attention partimage Icr whose feature information is deviated from the featureinformation of the photographed image Ic acquired at step S109 by adegree smaller than a predetermined threshold value, the positiondetermination unit 44 determines that the position indicated by thephotographed image Ic acquired at step S109 corresponds to the attentionpart.

Then, when the position (i.e., the shooting range of the photographedimage Ic) indicated by the photographed image Ic acquired at step S109corresponds to the landmark or the attention part (step S110; Yes), theoutput unit 45 performs an output prompting confirmation of thephotographed image Ic displayed on the display device 2 (step S111).Thereby, the inspection device 1 suitably suppresses the inspector fromoverlooking the inspection point. In some example embodiments, theoutput unit 45 further displays information on the landmark immediatelybefore or immediately after the attention part when it corresponds tothe attention part. It enables the inspector to recognize the relativeposition of the attention part with respect to the landmark.

On the other hand, when the position (i.e., the shooting range of thephotographed image Ic) indicated by the photographed image Ic acquiredat step S109 does not correspond the landmark nor the attention part(step S110; No), or after the completion of the step S111, theacquisition unit 41 determines whether or not the discharge of theendoscope 3 is completed (step S112). Then, when the acquisition unit 41determines that the discharge of the endoscope 3 is completed (stepS112; Yes), the acquisition unit 41 ends the process of the flowchart.On the other hand, when the acquisition unit 41 determines that thedischarge of the endoscope 3 is not completed (step S112; No), theacquisition unit 41 returns the process to the step S109.

(1-5) Modification

Next, a description will be given of modifications of the exampleembodiment described above. Subsequent modifications may be applied tothe example embodiments described above in combination.

(Modification 1-1)

The detection unit 42 of the inspection device 1 may execute the processof detecting the anomaly image Ica without using an auto encoder.

In this case, the detection unit 42 may perform detection of the anomalyimage Ica based on other general anomaly detection (i.e., an algorithmfor detecting an image indicating an anomalous object based on a groupof images indicating a target object in the normal state). For example,the detection unit 42 may detect the anomaly image Ica by using anunsupervised learning algorithm based on the PCA (Principal ComponentAnalysis), clustering or the k-nearest neighbor method or any otheralgorithm such as 1-class SVM which applies the support vector machine(SVM: Support Vector Machine) to the anomaly detection.

(Modification 1-2)

The parameter information storage unit 20, the landmark positioninformation storage unit 21, and the attention part information storageunit 22 may be stored in a storage device which is separated from theinspection device 1.

In this case, the inspection device 1 refers to and updates theparameter information storage unit 20, the landmark position informationstorage unit 21 and the attention part information storage unit 22 viathe interface 13. For example, when the above-described storage deviceis a server device or the like that communicates via a network, theinterface 13 includes a communication interface such as a networkadapter for performing communication. In addition, when theabove-described storage device is connected to the inspection device 1via a cable or the like, the interface 13 includes an interface thatconforms to USB, SATA (Serial AT Attachment) or the like, for example.Further, the parameter information storage unit 20, the landmarkposition information storage unit 21 and the attention part informationstorage unit 22 may be stored in a storage medium that can be referredto and updated by the inspection device 1.

(Modification 1-3)

The landmark position information storage unit 21 may store either oneof the anomaly image Ica and the feature information Ifa as the landmarkposition information, instead of storing both of them.

For example, in a case where the landmark position information does notinclude the anomaly image Ica, if the position determination unit 44determines that the photographed image Ic indicates the landmark, theoutput unit 45 performs an output prompting the confirmation withoutdisplaying the anomaly image Ica acquired in the insertion process ofthe endoscope 3. Further, in another case where the landmark positioninformation does not include the feature information Ifa, the positiondetermination unit 44 determines whether or not the photographed imageIc supplied from the acquisition unit 41 is similar to the anomaly imageIca included in the landmark position information. In this case, theposition determination unit 44 may make the above-mentioned similaritydetermination by generating the feature information of each of theimages by the auto encoder, or may make the above-mentioned similaritydetermination by any other general image similarity determinationmethod.

In the same way, the attention part information storage unit 22 mayinclude the feature information of the attention part image Icr as theattention part information instead of including the attention part imageIcr.

Second Example Embodiment

Next, a description will be given of a second example embodiment inwhich a partial image showing an anomalous cell is detected from partialimages of the pathological tissue image.

(2-1) Configuration

FIG. 7 illustrates the hardware configuration of the inspection device1A according to the second example embodiment. As shown in FIG. 7, theinspection device 1A includes a processor 11A, a memory 12A, and aninterface 13A.

The processor 11A executes a predetermined process by executing aprogram stored in the memory 12A. Examples of the processor 11A includea CPU and a GPU.

The memory 12A includes various memories such as RAM, ROM, and flashmemory. Further, the memory 12A stores a program for the inspectiondevice 1 to execute each process according to the second exampleembodiment. The memory 12A is also used as a working memory. The memory12A also includes a parameter information storage unit 20A that storesthe parameters of the auto encoder.

The interface 13A is an interface for performing an input process of thepathological tissue image Ix and an output process of a partial image(also referred to as “anomaly partial image Ixpa”) indicating ananomalous state from the pathological tissue image Ix. The interface 13Amay be a communication interface, such as a network adapter, forwirelessly or wirelessly transmitting and receiving data to and fromexternal devices under the control of the processor 11A. In anotherexample embodiment, the interface 13A may be an interface that conformsto USB or SATA for exchanging data with a storage device that is aperipheral device. The supply source of the pathological tissue image Ixand the supply destination of the anomaly partial image Ixpa may be thesame device or may be different devices.

The hardware configuration of the inspection device 1A is not limited tothe configuration shown in FIG. 7. For example, the inspection device 1Amay further include a display unit, a sound output unit, or/and an inputunit.

(2-2) Functional Block

FIG. 8 is a functional block diagram of an inspection device 1Aaccording to the second example embodiment. As shown in FIG. 8, theprocessor 11 of the inspection device 1A functionally includes anacquisition unit 41A and a detection unit 42A.

The acquisition unit 41A acquires the pathological tissue image Ix fromthe external device via the interface 13A. Here, the pathological tissueimage Ix has a very large number of pixels (e.g., vertical andhorizontal 7000 pixels). Then, the acquisition unit 41A generates apartial image (simply referred to as “partial image Ixp”) with apredetermined size (e.g., vertical and horizontal 256 pixels) from theacquired pathological tissue image Ix. For example, the acquisition unit41A generates a partial image Ixp for each cell nucleus included in thepathological tissue image Ix. Then, the acquisition unit 41A suppliesthe generated partial images Ixp to the detection unit 42A.

The detection unit 42A detects a partial image Ixp whose restorationerror is equal to or larger than a predetermined threshold value as ananomalous partial image Ixpa, wherein the restoration error is obtainedby inputting the partial image Ixp supplied from the acquisition unit41A to the auto encoder configured by referring to the parameterinformation storage unit 20A. Then, the detection unit 42A outputs thedetected anomaly partial image Ixpa to an external device or the like.Thereafter, for example, a more detailed abnormal inspection may beperformed for the anomaly partial image Ixpa by inputting the anomalypartial image Ixpa to an abnormal detection program with higher accuracyor an abnormal inspection by visual confirmation by the inspector may beperformed.

Here, the parameter information stored in the parameter informationstorage unit 20A is the information indicating the parameters of theauto encoder obtained by learning. In this case, the auto encoder islearned based on sample images for learning each of which indicates thestate of a normal cell nucleus. By referring to the parameterinformation generated in this way, the detection unit 42A can suitablyconfigure an auto encoder which has a small restoration error when thepartial image Ixp indicating the normal state is inputted thereto andwhich has a large restoration error when the partial image Ixpindicating the anomalous state is inputted thereto.

(2-3) Concrete Example

FIG. 9 is a display example of a pathological tissue image Ix indicatinga pathological tissue obtained by cutting the cells of the stomach. FIG.10 also illustrates pixel-by-pixel restoration errors outputted by theauto encoder for the pathological tissue image Ix. Specifically, FIG. 10illustrates the magnitude of the restoration error that is obtained bydividing the pathological tissue image Ix shown in FIG. 9 into partialimages Ixp and inputting each of the partial images Ixp to the autoencoder, wherein the auto encoder has been learned by use of only imagesindicating the cell parts of the normal stomach. In FIG. 10, forconvenience, the background of the cells is colored by a predeterminedcolor other than white, and the larger the restoration error is, thelighter (so as to approach white) the color of the tissue becomes.

In the pathological tissue image Ix shown in FIG. 9, the partssurrounded by the elliptical frames 51 to 53 correspond to the cancerparts. In this case, within the areas surrounded by the ellipticalframes 51A to 53A in FIG. 10 that are identical to the areas surroundedby the elliptical frame 51-53 of FIG. 9, the restoration error isclearly larger than the restoration error of the other cellular areas.Therefore, in this case, since the restoration error of the partialimages Ixp including the areas surrounded by the elliptical frames 51 to53 are equal to or larger than a predetermined threshold value, theinspection device 1A regards the partial images Ixp including the areassurrounded by the elliptical frames 51 to 53 as the anomalous partialimages Ixpa.

Thus, the inspection device 1A according to the second exampleembodiment can recognize the partial image(s) Ixp including the lesionpart such as a cancer as an anomaly partial image Ixpa. Then, theinspection device 1A can suitably detect and output the anomalouspartial image(s) Ixpa from the partial images Ixp generated from thepathological tissue image Ix.

(2-4) Processing Flow

FIG. 11 is a flowchart showing a process procedure of the inspectiondevice 1A according to the second example embodiment.

First, the acquisition unit 41A of the inspection device 1A acquires thepathological tissue image Ix from an external device or the like (stepS201). Next, the acquisition unit 41A generates a plurality of partialimages Ixp from the pathological tissue image Ix acquired at step S201(step S202).

Next, the detection unit 42A inputs a partial images Ixp into the autoencoder (step S203). Then, the detection unit 42A determines whether ornot the partial image Ixp corresponds to the anomaly partial image Ixpabased on the restoration error that is an error between the input to theauto encoder and the output of the auto encoder (step S204). When thepartial image Ixp corresponds to the anomaly partial image Ixpa (StepS204; Yes), the detection unit 42A outputs the partial image Ixp as theanomaly partial image Ixpa (Step S205). Meanwhile, when the partialimage Ixp does not correspond to the anomaly partial image Ixpa (stepS204; No), the detection unit 42A advances the process to the step S206.

Then, the detection unit 42A determines whether or not the determinationprocess at the step S204 has been performed for all the partial imagesIxp (step S206). Then, when the determination process of the step S204is performed for all the partial images Ixp (step S206; Yes), thedetection unit 42A ends the process of the flowchart. On the other hand,when the determination processing of the step S204 is not performed forall the partial images Ixp yet (step S206; No), the detection unit 42Areturns the process to the step S203.

(2-5) Modification

Next, a description will be given of preferred modifications to thesecond example embodiment. Subsequent modifications may be applied tothe second example embodiments described above in combination.

(Modification 2-1)

The detection unit 42A of the inspection device 1A may detect theanomaly partial image Ixpa based on the algorithm of the anomalydetection other than the auto encoder as with (Modification 1-1) of thefirst example embodiment described above.

(Modification 2-2)

The inspection device 1A may store the detected anomaly partial imageIxpa in the memory 12A of the inspection device 1A instead of outputtingit to a device other than the inspection device 1A. Similarly, theinspection device 1A may acquire the pathological tissue image Ix byreading out the pathological tissue image Ix stored in the memory 12A,instead of acquiring the pathological tissue image Ix from the externaldevice.

In addition, when the inspection device 1A stores an abnormalitydetection program capable of detecting abnormality with an accuracyhigher than the accuracy of the auto encoder, the inspection device 1Amay execute the above-described abnormality detection program for theanomaly partial image Ixpa. Even in this case, as compared with the caseof executing the abnormality detection program for all the partialimages Ixp, the inspection device 1A can perform efficient andhigh-precision abnormality detection thereby to suitably shorten theprocessing time.

In yet another example embodiment, the inspection device 1A may causethe inspector to visually recognize the anomaly partial image Ixpa bydisplaying the anomaly partial image Ixpa detected by the detection unit42A on the display unit and accept the input regarding the abnormalitydetermination by the inspector. Even in this case, the inspection device1A can limit the target of the visual abnormality determination by theinspector to the anomaly partial image(s) Ixpa that are a part of thepartial images Ixp.

(Modification 2-3)

The acquisition unit 41A of the inspection device 1A may acquire partialimages Ixp generated from the pathological tissue image Ix from theexternal device, instead of acquiring the pathological tissue image Ix.The acquisition unit 41A may acquire the partial images Ixp by readingout the partial images Ixp generated in advance and stored in the memory12A from the memory 12A.

(Modification 2-4)

The inspection device 1A may perform a process of determining whether ornot each partial image of the image used in the cytological examinationindicates the anomalous state, instead of the process of determiningwhether or not each partial image Ixp of the pathological tissue imageIx used in the pathological tissue diagnosis indicates an anomalousstate. Thus, even for the pathological image other than thephysiological tissue image Ix, the inspection device 1A executes theprocess according to the second example embodiment described abovethereby to suitably detect an image indicating an anomalous state.

Third Example Embodiment

FIG. 12 is a functional block diagram of an inspection device 1Baccording to the third example embodiment. The inspection device 1Bfunctionally includes an acquisition unit 41B and a detection unit 42B.

The acquisition unit 41B acquires a target image indicating a targetobject of inspection. For example, the target image may be aphotographed image Ic according to the first example embodiment or maybe a partial image Ixp according to the second example embodiment. Thedetection unit 42B detects, on the basis of a group of images each ofwhich indicates the target object in a normal state, the target imagethat indicates the target object that is not in the normal state amongthe target images that the acquisition unit 41B acquires. In this case,for example, in the same way as the first and second exampleembodiments, the detection unit 42B detects the target image whichindicates the target object that is not in the normal state on the basisof the auto encoder learned by use of the group of the images each ofwhich indicates the target object in the normal state.

Even according to the third example embodiment, the inspection device 1Bcan suitably detect the target image which indicates the target objectof inspection that is not in the normal state.

The whole or a part of the example embodiments described above(including modifications, the same applies hereinafter) can be describedas, but not limited to, the following Supplementary Notes.

[Supplementary Note 1]

An inspection device comprising:

an acquisition unit configured to acquire a target image which indicatesa target object of inspection; and

a detection unit configured to detect, on a basis of a group of imageseach of which indicates the target object in a normal state, the targetimage that indicates the target object that is not in the normal stateamong the target images that the acquisition unit acquires.

[Supplementary Note 2]

The inspection device according to Supplementary Note 1,

wherein, on a basis of an auto encoder learned by use of the group ofthe images, the detection unit detects the target image that indicatesthe target object that is not in the normal state among the targetimages that the acquisition unit acquires.

[Supplementary Note 3]

The inspection device according to Supplementary Note 1 or 2,

wherein the acquisition unit acquires, as the target image, an imagephotographed in time series by a photographing unit which is insertedinto a lumen that is the target object in endoscopy.

[Supplementary Note 4]

The inspection device according to Supplementary Note 3, furthercomprising

a position information generation unit configured to generate, in aninsertion process of the photographing unit, information, which includesat least one of the target image detected by the detection unit orfeature information thereof, as landmark position information whichindicates a position of a landmark in the lumen.

[Supplementary Note 5]

The inspection device according to Supplementary Note 4,

wherein the position information generation unit generates, in theinsertion process of the photographing unit, information, which includesat least one of the target image acquired by the acquisition unit at atime of detecting external input by an inspector or feature informationthereof, as attention part information which indicates an attention partin the lumen.

[Supplementary Note 6]

The inspection device according to Supplementary Note 4 or 5,

wherein the position information generation unit generates, in theinsertion process of the photographing unit, information, which includesat least one of the target image detected as an inspection targetthrough an image analysis or feature information thereof, as attentionpart information which indicates an attention part in the lumen.

[Supplementary Note 7]

The inspection device according to Supplementary Note 5 or 6,

wherein the position information generation unit stores the attentionpart information in a storage unit in association with at least one ofthe landmark position information generated immediately beforegeneration of the attention part information or the landmark positioninformation generated immediately after the generation of the attentionpart information.

[Supplementary Note 8]

The inspection device according to any one of Supplementary Notes 4 to7, further comprising

an output unit configured to output information prompting conformationof the target image acquired in a discharge process of the photographingunit by the acquisition unit, in a case where the target imagecorresponds to the landmark position information.

[Supplementary Note 9]

The inspection device according to any one of Supplementary Notes 5 to7, comprising

an output unit configured to output information prompting conformationof the target image acquired in a discharge process of the photographingunit by the acquisition unit, in a case where the target imagecorresponds to the attention part position information.

[Supplementary Note 10]

The inspection device according to Supplementary Note 1 or 2,

wherein the acquisition unit acquire partial images generated from apathological image as the target image.

[Supplementary Note 11]

An inspection method executed by an inspection device, the inspectionmethod comprising:

acquiring a target image which indicates a target object of inspection;and

detecting, on a basis of a group of images each of which indicates thetarget object in a normal state, the target image that indicates thetarget object that is not in the normal state among the acquired targetimages.

[Supplementary Note 12]

A program executed by a computer, the program causing the computer tofunction as:

an acquisition unit configured to acquire a target image which indicatesa target object of inspection; and

a detection unit configured to detect, on a basis of a group of imageseach of which indicates the target object in a normal state, the targetimage that indicates the target object that is not in the normal stateamong the target images that the acquisition unit acquires.

While the invention has been particularly shown and described withreference to example embodiments thereof, the invention is not limitedto these embodiments. It will be understood by those of ordinary skilledin the art that various changes in form and details may be made thereinwithout departing from the spirit and scope of the present invention asdefined by the claims. In other words, it is needless to say that thepresent invention includes various modifications that could be made by aperson skilled in the art according to the entire disclosure includingthe scope of the claims, and the technical philosophy. All PatentLiteratures mentioned in this specification are incorporated byreference in its entirety.

DESCRIPTION OF REFERENCE NUMERALS

-   -   1 Inspection equipment    -   2 Display device    -   3 Endoscope    -   11, 11A processor    -   12, 12A memories    -   13, 13A interface    -   14 Input unit    -   15 Light source unit    -   100 Endoscopic inspection system

What is claimed is:
 1. An inspection device comprising a processorconfigured to: acquire a target image which indicates a target object ofinspection; and detect, on a basis of a group of images each of whichindicates the target object in a normal state, the target image thatindicates the target object that is not in the normal state among theacquired target images.
 2. The inspection device according to claim 1,wherein, on a basis of an auto encoder learned by use of the group ofthe images, the processor detects the target image that indicates thetarget object that is not in the normal state among the acquired targetimages.
 3. The inspection device according to claim 1, wherein theprocessor acquires, as the target image, an image photographed in timeseries by a photographing unit which is inserted into a lumen that isthe target object in endoscopy.
 4. The inspection device according toclaim 3, wherein the processor is further configured to generate, in aninsertion process of the photographing unit, information, which includesat least one of the detected target image or feature informationthereof, as landmark position information which indicates a position ofa landmark in the lumen.
 5. The inspection device according to claim 4,wherein the processor generates, in the insertion process of thephotographing unit, information, which includes at least one of thetarget image acquired at a time of detecting external input by aninspector or feature information thereof, as attention part informationwhich indicates an attention part in the lumen.
 6. The inspection deviceaccording to claim 4, wherein the processor generates, in the insertionprocess of the photographing unit, information, which includes at leastone of the target image detected as an inspection target through animage analysis or feature information thereof, as attention partinformation which indicates an attention part in the lumen.
 7. Theinspection device according to claim 5, wherein the processor stores theattention part information in a storage unit in association with atleast one of the landmark position information generated immediatelybefore generation of the attention part information or the landmarkposition information generated immediately after the generation of theattention part information.
 8. The inspection device according to claim4, wherein the processor is further configured to output informationprompting conformation of the target image acquired in a dischargeprocess of the photographing unit, in a case where the target imagecorresponds to the landmark position information.
 9. The inspectiondevice according to claim 5, wherein the processor is configured tooutput information prompting conformation of the target image acquiredin a discharge process of the photographing unit, in a case where thetarget image corresponds to the attention part position information. 10.The inspection device according to claim 1, wherein the processoracquire partial images generated from a pathological image as the targetimage.
 11. An inspection method executed by an inspection device, theinspection method comprising: acquiring a target image which indicates atarget object of inspection; and detecting, on a basis of a group ofimages each of which indicates the target object in a normal state, thetarget image that indicates the target object that is not in the normalstate among the acquired target images.
 12. A non-transitorycomputer-readable storage medium storing a program executed by acomputer, the program causing the computer to: acquire a target imagewhich indicates a target object of inspection; and detect, on a basis ofa group of images each of which indicates the target object in a normalstate, the target image that indicates the target object that is not inthe normal state among the acquired target images.